Crowd-Sourced Iterative Annotation for Narrative Summarization Corpora
نویسندگان
چکیده
We present an iterative annotation process for producing aligned, parallel corpora of abstractive and extractive summaries for narrative. Our approach uses a combination of trained annotators and crowd-sourcing, allowing us to elicit human-generated summaries and alignments quickly and at low cost. We use crowd-sourcing to annotate aligned phrases with the text-to-text generation techniques needed to transform each phrase into the other. We apply this process to a corpus of 476 personal narratives, which we make available on the Web.
منابع مشابه
An Active Analysis and Crowd Sourced Approach to Social Training
Interactive narrative (IN) has increasingly been used for social skill training. However, extensive content creation is needed to provide learners with flexibility to replay scenarios with sufficient variety to achieve proficiency. Such flexibility requires considerable content creation appropriate for social skills training. The goal of our work is to address these issues through developing a ...
متن کاملA Prototype Tool Set to Support Machine-Assisted Annotation
Manually annotating clinical document corpora to generate reference standards for Natural Language Processing (NLP) systems or Machine Learning (ML) is a timeconsuming and labor-intensive endeavor. Although a variety of open source annotation tools currently exist, there is a clear opportunity to develop new tools and assess functionalities that introduce efficiencies into the process of genera...
متن کاملMinimal Narrative Annotation Schemes and Their Applications
The increased use of large corpora in narrative research has created new opportunities for empirical research and intelligent narrative technologies. To best exploit the value of these corpora, several research groups are eschewing complex discourse analysis techniques in favor of high-level minimalist narrative annotation schemes that can be quickly applied, achieve high inter-rater agreement,...
متن کاملA Methodology for Corpus Annotation through Crowdsourcing
In contrast to expert-based annotation, for which elaborate methodologies ensure high quality output, currently no systematic guidelines exist for crowdsourcing annotated corpora, despite the increasing popularity of this approach. To address this gap, we define a crowd-based annotation methodology, compare it against the OntoNotes methodology for expert-based annotation, and identify future ch...
متن کاملCrowd-sourcing and author submission as alternatives to professional curation
Can we decrease the costs of database curation by crowd-sourcing curation work or by offloading curation to publication authors? This perspective considers the significant experience accumulated by the bioinformatics community with these two alternatives to professional curation in the last 20 years; that experience should be carefully considered when formulating new strategies for biological d...
متن کامل